Search for: All records

In making validity arguments, a central consideration is whether the instrument fairly and adequately covers intended content, and this is often evaluated by experts. While common procedures exist for quantitatively assessing this, the effect of loss aversion—a cognitive bias that would predict a tendency to retain items—on these procedures has not been investigated. For more novel constructs, experts are typically drawn from adjacent domains. In such cases, a related cognitive bias, the ownership effect, would predict that experts would be more loss averse when considering items closer to their domains. This study investigated whether loss aversion and the ownership effect are a concern in standard content validity evaluation procedures. In addition to including promising items to measure a relatively novel construct, framing agency, we included distractor items linked to other areas of our evaluators’ expertise. Experts evaluated all items following procedures outlined by Lawshe (1975). We found on average, experts were able to distinguish between the intended items and distractor items. Likewise, on average, experts were somewhat more likely to reject distractor items closer to their expertise. This suggests that loss aversion and the ownership effect are not likely to bias content validation procedures. 

Background: Because of prior experience solving well-structured problems that have single, correct answers, students often struggle to direct their own design work and may not understand the need to frame ill-structured design problems. Purpose: Framing agency—defined as making decisions that are consequential to framing design problems and learning through this process—sheds light on students’ treatment of design problems; by framing, we mean the various actions designers take to understand, define, and bound the problem. Using the construct framing agency, we sought to characterize design team discourse to detect whether students treated design problems as ill- or well-structured and examine the consequences of this treatment. Method: Data were collected through extended participant observation of a capstone design course in a biomedical engineering program at a large research university. Data included audio and video records of design team meetings over the course of framing and solving industry-sponsored problems. For this paper, we analyzed three cases using sociolinguistic content analysis to characterize framing agency and compared the cases to illuminate the nuances of framing agency. Results: All teams faced impasses; one team navigated the impasse by framing the problem, whereas the others treated the problem as given. We identified markers of agency in students’ discourse, including tentative language, personal pronouns, and sharing ownership. Conclusions: Framing agency clarifies the kinds of learning experiences students need in order to overcome past experiences dominated by solving archetypical well-structured problems with predetermined solutions.